RAKE stop list generation

Stop list generation for [RAKE

Words adjacent to but not included in the keyword are good candidates for [stopword

Both precision and recall were improved by excluding from the stop list words whose frequency in the keyword is higher than the frequency adjacent to the keyword.

F value is best with the largest stop list and could be made even better by making it larger.

The stop list made by DF alone is worse than the opposite.

Interesting to use DF instead of TF.

impressions

I see what you mean

I don't like the idea of splitting it into two values in a snap based on "more or less," but that's not the problem with this algorithm, it's the problem with the concept of stopwords in the first place.

Extending to the domain of probability, we can say that when a word is observed, the probability that it is part of a keyword and the probability that it is adjacent to a keyword

If a keyword is at the edge of a candidate, it is reasonable to think about whether to expand the candidate or not, and to say, "Let's not expand it because the probability of it being adjacent is higher.

Not when it's among the keyword candidates, right?

For example, "of" and "of" have a low probability of being present at the middle end of keyword candidates, but they are rather common in some of them.

So if it's overdivided, if it comes up more than once, it's rejoined, but I don't know if it's a good balance...

Each word has a "probability of being among the keyword candidates".

The stopword is it's zero, and the state it approximates.

Scores to be multiplied

Score for grammatical appropriateness of the keyword form

The score of a token sequence is calculated as the product of the scores of each token

State that this is 0/1 depending on whether this is a stop list element or not.

If it is a probability value, the more you multiply, the smaller it gets.

→Bias for selection of short key phrases

Offset by [Bias for longer key phrases to be selected

If you have a "document with some keywords already marked up" like Scrapbox, you can get a stop list from there.

---